An improved prognostic model for predicting the mortality of critically ill patients: a retrospective cohort study

A simple prognostic model is needed for ICU patients. This study aimed to construct a modified prognostic model using easy-to-use indexes for prediction of the 28-day mortality of critically ill patients. Clinical information of ICU patients included in the Medical Information Mart for Intensive Care III (MIMIC-III) database were collected. After identifying independent risk factors for 28-day mortality, an improved mortality prediction model (mionl-MEWS) was constructed with multivariate logistic regression. We evaluated the predictive performance of mionl-MEWS using area under the receiver operating characteristic curve (AUROC), internal validation and fivefold cross validation. A nomogram was used for rapid calculation of predicted risks. A total of 51,121 patients were included with 34,081 patients in the development cohort and 17,040 patients in the validation cohort (17,040 patients). Six predictors, including Modified Early Warning Score, neutrophil-to-lymphocyte ratio, lactate, international normalized ratio, osmolarity level and metastatic cancer were integrated to construct the mionl-MEWS model with AUROC of 0.717 and 0.908 for the development and validation cohorts respectively. The mionl-MEWS model showed good validation capacities with clinical utility. The developed mionl-MEWS model yielded good predictive value for prediction of 28-day mortality in critically ill patients for assisting decision-making in ICU patients.


Methods
Study design. This study analyzed a retrospective cohort of patients admitted to the ICU (aged 14 years or older). A new MEWS scoring system was developed with the aim of better predicting the 28-day all-cause mortality of critically ill patients with a validation display in a nomogram. The datasets used in this study were derived from the publicly available database MIMIC-III (version 1.4), which contains high-quality health-related data from patients who were admitted to the ICU of the Beth Israel Deaconess Medical Center between 2001 and 2012. After completing the National Institutes of Health web-based training course, we obtained approval to access the database (Certification Number: 37764466). Informed consent was not required because all protected health information had been de-identified.
Study population. We reviewed the discharge summaries of all patients in the MIMIC-III database admitted to the ICU between 2001 and 2012. All ICU patients aged > 14 years old with a measured MEWS within 24 h after ICU admission were included in this study. Patients who met any of the following criteria were excluded: (1) age less than 14 years; (2) readmission in the same hospitalization (only data from the first ICU admission were included in forming the final cohort); (3) unavailability of MEWS due to omission of a measurement within 24 h after ICU admission. The screened ICU patients were eligible for subsequent analysis.
Data extraction, management, and processing. Demographic, clinical, and laboratory data and risk scoring results were extracted with structured query language using PostgreSQL tools (version 9.6) or calculated from the following tables: ADMISSIONS, ICUSTAYS, CHARTEVENTS, DIAGNOSIS_ICD, d_items, d_icd_diagnoses, LABEVENTS, PATIENTS, prescriptions, and Materialized Views. The extracted items for demographic, clinical, and laboratory data and risk scoring results in the database are listed in Table 2. The data processing, including missing data imputation and Winsorizing, was only performed on the development set, and the validation set was used to validate the predictive performance of the developed model. The worst values for lab parameters were selected if they were measured multiple times within 48 h before and after ICU admission. The body mass index (BMI) was calculated as weight (kg)/height (m) 2 , and osmolarity was calculated as (2 × sodium + potassium) + (glucose/18) + (blood urea nitrogen/2.8). The risk scoring systems including the APACHE-II and MEWS. The APACHE II scoring system is based on 12 physiological variables (temperature, mean arterial pressure, heart rate, respiratory rate, alveolar-arterial oxygen tension difference [fraction of inspired oxygen (FiO 2 > 50%)] or partial pressure of oxygen in the arterial blood (PaO 2 ; FiO 2 < 50%), arterial pH or HCO 3 , serum sodium, serum potassium, serum creatinine, hematocrit, white blood cell count, and the Glasgow Coma Scale score, a chronic health evaluation and age adjustment score. Each variable was calculated using the worst values for these parameters recorded during the first 24 h in the ICU; the range for the total Apache II score is 0-71 points. The APACHE-II scores for all ICU patients were acquired with the SQL code from the Materialized Views of the MIMIC-III database. The MEWS was calculated according to Table 3 7 with the worst  values within 24 h after ICU admission selected for the parameter used for the MEWS evaluation. Because the true ages of patients over 89 years old were omitted due to the privacy policy of the MIMIC database, we selected age × 90/300 as a surrogate age for those patients. In data processing, we used multiple k-Fold cross validation of the mionl-MEWS score. We performed k-fold cross-validation with five random folds for the total of 51,121 patients. We compared the AUROC, positive predictive value (PPV), and negative predictive value (NPV) values between the model and cross-validation to show the robustness of our model.

Nomogram development for the simplified prediction model. A nomogram is a graphical tool that
can be easily used by clinicians in a resource-limited environment, as no statistical software or online electronic calculator is required. In this study, a nomogram was formulated with clinical practicability based on the results for the obtained predictive model.

Statistical analysis.
All patients were divided into two cohorts (development vs. validation) with complete randomization. The distributions of continuous variables were assessed by the Kolmogorov-Smirnov test, and data with skewed distributions were log normalized. Normally distributed continuous variables were expressed as mean ± standard deviation (SD), and non-normally distributed continuous variables were expressed as median (interquartile range). Categorical variables were expressed as absolute values (percentages). Descriptive statistics from the development and validation cohorts were used to compare the baseline data between survivors and nonsurvivors with the t test for normally distributed data, the Mann-Whitney U test for non-normally distributed data, and the chi-squared test for categorical variables. The covariates associated with 28-day allcause mortality were further identified with univariate and multivariate logistic regression analyses. For each variable, the unadjusted and adjusted odds ratios (ORs) were assessed and reported with p-values and 95% CIs.
The multivariate logistic regression model (mionl-MEWS) was built using a forward selection modeling process with a significance level of 0.05. The variables independently associated with 28-day mortality (metastatic cancer, MEWS, lac concentration, NLR, INR, and osmolarity level) were included in the final model. Furthermore, potential multicollinearity was tested using a mean variance inflation factor (VIF), where a value ≥ 10 indicated multicollinearity. Additionally, we assessed the discriminative abilities of the different models based on AUROC values. We then applied the obtained model generated from the development dataset to the validation dataset and assessed the discriminative ability based on the AUROC and the calibration capacity based on the H/L C-statistic. We also generated the calibration curves and calculated the Brier scores for predicting mortality among both the development and validation cohorts. The PR-AUC was applied to evaluate the predictive performance considering clinical application with the validation cohort.   Fig. 1. Hypertension (54.68%) was the most common comorbidity, followed by cardiac arrhythmia (30.00%), diabetes (28.15%), and congestive heart failure (28.05%). In our study, 34,081 patients (66.67%) were randomly assigned to the development cohort, and 17,040 patients (33.33%) were assigned to the validation cohort.

Development of a risk prediction model for 28-day all-cause mortality of ICU patients. The
28-day all-cause mortality percentages among critically ill patients were 13.39% in the development cohort (4069/30,399) and 13.61% in the validation cohort (2319/17,040). Significant differences in baseline clinical features, risk scores, and laboratory data were observed between survivors and nonsurvivors, as summarized in Tables 4 and 5. In the development cohort, nonsurvivors were predominantly male and compared with survivors, they had a significantly higher incidence of chronic medical conditions or comorbidities such as congestive heart failure, cardiac arrhythmia, pulmonary circulation disease, vasoactive drug use, liver disease, renal failure, hypothyroidism, paralysis, other neurological disease, solid tumor, metastatic cancer, lymphoma, and coagulopathy; and had a significantly lower incidence of valvular disease, hypertension, diabetes, psychoses, depression, and alcohol or drug abuse. Compared with survivors, nonsurvivors also were older and had significantly higher values for length of ICU stay, APACHE-II score, MEWS, white blood cell count, RDW, NLR, platelet (PLT) count, total bilirubin level, INR, aspartate transaminase level, alanine transaminase level, prothrombin time, activated partial thromboplastin time, blood urea nitrogen level, serum creatinine level, blood glucose level, lac concentration, osmolarity, and sodium level. In addition, the nonsurvivors had significantly lower hemoglobin, pH, PaO 2 , PaCO 2 , SO 2 , and PaO 2 /FiO 2 values compared with the survivors. No significant differences in the rates of peripheral vascular disease, chronic pulmonary disease, peptic ulcer, anemia due to blood loss, deficiency anemia, rheumatoid arthritis, or acquired immune deficiency syndrome were observed between survivors and nonsurvivors in the development cohort. However, significant differences in the incidence rates like congestive heart failure, valvular disease, peripheral vascular disease, anemia due to blood loss, rheumatoid arthritis, and acquired immune deficiency syndrome etc. were also observed between the survivors and nonsurvivors in the validation cohort. The baseline characteristics showed similar distributions between the development and validation cohorts, indicating the successful randomization in the present study.  www.nature.com/scientificreports/ Next, we included the variables that differed significantly between survivors and nonsurvivors of the development cohort in univariate logistic regression analysis. The results presented in Table 6 demonstrated that all selected variables were significantly associated with 28-day mortality in the univariate logistic regression analysis, similar to the results of the abovementioned univariate analyses. The demographic characteristics with the three largest OR values were: age, OR = 1.033, p < 0.001; BMI, OR = 0.995, p < 0.001; and sex, OR = 0.880, p < 0.001. The three chronic medical conditions or comorbidities with the largest OR values were: metastatic cancer, OR = 2.833, p < 0.001; coagulopathy, OR = 2.100, p < 0.001; and requirement of vasoactive drug therapy, OR = 1.812, p < 0.001. For risk scores and laboratory parameters, we selected the indicators with a low cost and a high frequency of use in the ICU. For example, the MEWS can be obtained by simple calculation with the parameters on the nursing record sheet; the RDW and NLR can be obtained via routine blood tests; and the lac concentration, INR, and osmolarity can be obtained using portable testing tools. Regarding the lac concentration, INR, RDW, NLR, and osmolarity, significantly increasing 28-day mortality rates were observed in patients with a lower BMI or a higher www.nature.com/scientificreports/ Considering that this predictive model was constructed based on the MEWS, NLR, lac concentration, INR, osmolarity level, and presence of metastatic cancer, the model was named the "mionl-MEWS" model. The AUROC for 28-day mortality using the mionl-MEWS for critically ill patients was 0.717 (95% CI 0.708-0.726, p < 0.001). The calculated H/L C-statistic was equal to 11.27 (p = 0.187), and the calibration plot of the observed versus expected probabilities for assessment of model performance is displayed in Fig. 3. The AUROC values for the APACHE-II, MEWS, RDW, NLR, lac concentration, and osmolarity were 0.743, 0.667, 0.639, 0.603, 0.594, and 0.622, respectively (Table 8). Statistical differences were detected among these AUROC (p < 0.001; Fig. 4 (Table 8). Similarly, statistical differences were also detected among these AUROC values (p < 0.001).
Although the AUROC for the mionl-MEWS appeared to be greater than that for the APACHE-II, the difference was not found to be significant (p = 0.120; Fig. 6). The PR-AUCs for the mionl-MEWS and APACHE-II were 0.907 and 0.899, respectively (Fig. 7). The Brier scores were as follows:  Table 8). These results indicate that the mionl-MEWS had good predictive ability with great calibration abilities. Importantly, the mionl-MEWS was not found to be inferior to the APACHE-II and was shown to be superior to other risk scores in the validation group.

k-Fold cross validation of the mionl-MEWS score.
To further illustrate the robustness of the developed mionl-MEWS model, we used repetitive randomization and k-fold cross validation (k = 5) to analyze the total of 51,121 patients. The AUROC for our model was 0.898 and that with k-fold cross-validation was 0.895  www.nature.com/scientificreports/   Nomogram for the mionl-MEWS score. Because the AUROC value provides limited information regarding how a prediction score works in clinical practice, a nomogram is needed to visualize the prognostic model for clinicians, and this graph is useful in resource-limited settings such as those without statistical software or electronic calculators. We translated the model with integrated independent factors into a nomogram using Stata statistical software. The prognostic nomogram derived from the mionl-MEWS score for clinical application is shown in Fig. 9.

Discussion
To the best of our knowledge, this retrospective study is the first to propose a simple prognostic model (mionl-MEWS) combining metastatic cancer, MEWS, lac, NLR, INR, and osmolarity level for the prediction of 28-day mortality in critically ill patients with internal validation. Based on the AUROC and PR-AUC values, the predictive efficacy of the mionl-MEWS for 28-day mortality in critically ill patients was superior to that of the traditional MEWS, NLR, RDW, lac, or osmolarity alone. Hence, the mionl-MEWS could be used to assist with clinical decision-making in the management of ICU patients.
Considering the likelihood of long in-hospital stays and high medical resource consumption, early identification of mortality risk using prognostic scoring systems is important for timely and effective management and intervention in critically ill patients in the ICU. In addition, patterns of ICU admissions have changed due to advances in the treatment of solid malignancies with immunotherapy and targeted therapies. For example, the proportion of patients with metastatic diseases increased from 48.6% in 2007-2008 to 60.2% in 2017-2018 in France 18 . Although many scoring systems for critical illness have been proposed to translate the complexity of patients' conditions into a single measure based on quantitative survival probabilities in current clinical practice, the drawbacks and flaws of these individual systems cannot be ignored. For instance, some assessment tools require many blood tests and/or scoring items, which can be time-consuming and lead to delayed interventions and/or a high financial burden for patients. Thus, fast, convenient, and inexpensive evaluation tools are needed in clinical practice. www.nature.com/scientificreports/ Our study retrospectively collected variables that could predict the 28-day mortality in critically ill patients. These variables, such as the MEWS, lac, NLR, INR, etc., were chosen from the literature and used in previous ICU risk assessment models. In our study, we demonstrated that compared with survivors, nonsurvivors tended to be older; male; have a higher incidence of metastatic cancer, coagulopathy, and vasopressor drug use within 48 h; have a lower BMI; and have higher MEWS, RDW, NLR, lac, INR, and osmolarity values, indicating that these factors might serve as potential prognostic markers in critically ill patients. Next, we investigated the factors that independently predicted 28-day mortality in critically ill patients. Our initial multivariate logistic regression analysis also showed that age, metastatic cancer, coagulopathy, MEWS, lac concentration, NLR, RDW, INR, and osmolarity level were independent predictors for 28-day mortality. Unfortunately, multicollinearity was detected among age, RDW, and osmolarity level. However, a series of studies have demonstrated that RDW has predictive value for mortality in patients with heart failure, septic shock, acute respiratory distress syndrome, etc. 14,19,20 . In addition, age ≥ 80 years was shown to be associated with higher ICU and hospital death compared with younger ages 21 . In our study, RDW and age also showed a correlation with the mortality of critically ill patients (OR 1.512; 95% CI 1.377-1.660; OR 1.038; 95% CI 1.036-1.041, p < 0.001, respectively). Nevertheless, in a previous cohort study of 8089 individuals analyzing the effect of age and RDW, the age-dependency of RDW seemed to be a universal biological feature 22 . Therefore, we removed age and RDW from our model to avoid multicollinearity in subsequent modeling.
Among the three underlying disease variables, metastatic cancer was previously shown to be an important predictor of a high 30-day mortality in the ICU 23 along with mechanical ventilation and vasopressor use 24 .
In the present study, the OR value for metastatic cancer as a predictor of 28-day mortality was 2.791 (95% CI 2.474-3.150; p < 0.001), which is similar to that reported by Barth et al. for the outcome of patients with metastatic lung cancer admitted to the ICU (OR 4.22 [1.4-12.4]; p = 0.008) 24 . Therefore, tumor metastasis should be considered in the decision-making process in the ICU. Coagulopathy also is a common cause for a poor prognosis in critically ill patients in the ICU, and its severity has been shown to predict hospital mortality standardized by INR 25 . Therefore, we only selected INR for inclusion in the final model analysis. Finally, vasopressors are commonly administered to ICU patients with hypotension to raise patients' blood pressure 26 . Decision-making regarding the timing of vasopressor initiation as well as balancing the risks and benefits of vasopressor use remains challenging. In the dataset used in our study, the proportion of patients who required treatment with a vasopressor within 48 h was significantly higher in the nonsurvivor group than in the survivor group (27.8% vs. 17.52% p < 0.001). Interestingly, vasopressor use was not found to be an influencing factor in our multiple regression analysis though. In a cohort study regarding the mortality of septic shock patients, only a weak correlation between the timing of vasopressor initiation and hospital mortality was found (adjusted OR 1.02, 95% CI 1.01-1.03, p < 0.001) 27 . These results also indirectly corroborate the finding in the present study that the timing of vasopressor initiation might not be associated with 28-day mortality in the ICU.
Among the indexes, MEWS was developed as a practical tool that can rapidly and effectively estimate clinical death risk using only five simple and basic physiological parameters without increasing the economic burden, since these parameters can be acquired from patient's electronic medical records automatically. In a previous observational study, Moon et al. found that the introduction of MEWS charts significantly reduced the number of in-hospital cardiac arrest calls (2% vs. 3%; p = 0.004) and in-hospital mortality rates (42% vs. 52%; p = 0.05) 28 . In addition, in a study predicting the 28-day mortality rate of ICU patients with severe septic shock, the MEWS was associated with the 28-day mortality rate (OR 1.462; 95% CI 1.122-1.905; p = 0.005) 29 , which was consistent with the finding in our study (OR 1.250; 95% CI 1.232-1.269; p < 0.001). However, another study found that the MEWS had limited ability to estimate sudden disease aggravation in patients, such as the occurrence of cardiac shock 30 . Therefore, the predictive value of the MEWS alone for the mortality rate in critically ill patients required further investigation.
Sepsis is well-recognized major health problem in the ICU globally. One study found that the proportion of ICU patients with ICU-acquired sepsis was 24.4% and that the mortality of hospitalized sepsis patients was very high at 25-30% 31 . Whether patients had sepsis was an important factor affecting the mortality of ICU patients. NLR, as an immune-related biomarker, was shown to serve as a convenient prognostic marker in sepsis patients. In their study predicting 28-day mortality in sepsis patients, Liu et al. reported that the NLR was associated with the 28-day mortality rate (OR 1.340; 95% CI 1.253-1.434; p < 0.001) 32 . However, in the present study, the OR value for the NLR was only 1.045 (95% CI 1.038-1.053; p < 0.001). This consistency might be due to differences in the study populations, as Liu et al. only selected patients with sepsis, and the present population was based on all ICU patients, not only those with sepsis. Previously, the lac concentration has been associated with mortality in different groups of critically ill patients, such as those with cardiogenic, hypovolemic, or septic shock. Relative hyperlactatemia (1.36-2.00 mmol/L) within the first 24 h of ICU admission was reported to be an independent predictor for in-hospital and ICU mortality in critically ill patients 16 . In addition, osmolarity with a threshold of 300 mmol/L was shown to be associated with increased mortality in critically ill patients with cardiac, cerebral, vascular, or gastrointestinal diagnoses at admission 33 , and these findings are consistent with those of our study (OR 1.669; 95% CI 1.517-1.836; p < 0.001).
Due to the complexity and heterogeneity in disease among critically ill patients, combination of different indexes can more accurately reflect the prognosis of ICU patients than any single index 34 . Thus, we included metastatic cancer, MEWS, lac concentration, NLR, INR, and osmolarity level in our model to predict 28-day mortality. Using the ROC curves to evaluate the 28-day mortality of critically ill patients, a higher AUROC values in the development cohort (0.717) and the validation cohort (0.908) were found upon combination of these six parameters as a composite index compared with each parameter separately. Notably, the mionl-MEWS had the greatest AUROC value, superior to those of the MEWS, RDW, osmolarity, NLR, and lac alone, indicating that the mionl-MEWS can provide a more comprehensive reflection of each patient's condition from six dimensions, including metastatic cancer for the distribution characteristics of ICU patients, MEWS for patients' general www.nature.com/scientificreports/ condition, lac concentration for microcirculation, NLR for sepsis, INR for coagulopathy, and osmolarity for the internal environment. Furthermore, we used the Brier score to assess the accuracy of our developed model. Among the evaluated indexes, the mionl-MEWS had the smallest Brier score in the development cohort and the third lowest score in the validation cohort, indicating that the mionl-MEWS offered good accuracy for prediction at an individual level. Additionally, we calculated the H/L C-statistic to assess consistent agreement between the observed ICU mortality and the actual ICU mortality. The mionl-MEWS showed adequate calibration, suggesting the assignment of the correct probability at all levels of predicted risk. Finally, the mionl-MEWS model provided stable evaluation with excellent calibration in the validation group (AUROC: 0.908 and PR-AUC: 0.907).
Our study has some strengths. First, to our knowledge, this study is the first to demonstrate enhanced prognostic ability for 28-day mortality in ICU patients via the combination of metastatic cancer, MEWS, lac concentration, NLR, INR, and osmolarity level. Second, the sample size in our study was relatively large, which reduced selection bias. Furthermore, we applied different probability models to evaluate the mionl-MEWS model in order to ensure the scientific nature and credibility of the results. Third, the parameters included in the mionl-MEWS model are objective and easily accessible among laboratory parameters that are widely available to clinicians. Fourth, the constructed nomogram makes 28-day mortality prediction easy and rapid in clinical practice.
Nevertheless, it is important to recognize the limitations of our study. Our data were collected retrospectively from the MIMIC-III database, and because this was a single-center retrospective study, it might be difficult to extend the findings to other hospitals. External validation in cohorts from other countries is needed to generalize our findings. Additionally, due to incomplete data collection and inaccurate data elements from the MIMIC-III database, the potential for bias cannot be excluded.

Conclusion
In the present study, we developed a prediction model, the mionl-MEWS, for the 28-day mortality of critically ill patients admitted to the ICU, demonstrated internal validation, and ensured the included clinical variables can be easily obtained in resource-limited settings. Our results showed that the mionl-MEWS offered higher predictive value for the 28-day mortality of critically ill patients compared with other scoring variables and/or systems. However, additional research is required to demonstrated whether the mionl-MEWS can be applied widely and extensively.